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Abstract 

The second-order achievable asymptotics in typical random number generation problems such 
as resolvability, intrinsic randomness, fixed-length source coding are considered. In these problems, 
several researchers have derived the first-order and the second-order achievability rates for general 
sources using the information spectrum methods. Although these formulas are general, their 
computation are quite hard. Hence, an attempt to address explicit computation problems of 
achievable rates is meaningful. In particular, for i.i.d. sources, the second-order achievable rates 
have earlier been determined simply by using the asymptotic normality. In this paper, we consider 
mixed sources of two i.i.d. sources. The mixed source is a typical case of nonergodic sources and 
whose self-information docs not have the asymptotic normality. Nonetheless, we can explicitly 
compute the second-order achievable rates for these sources on the basis of two-peak asymptotic 
normality. In addition, extensions of our results to more general mixed sources, such as a mixture 
of countably infinite i.i.d. sources or Markovian sources, and a continuous mixture of i.i.d. sources, 
are considered. 
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Second-Order Achievability, Random Number Generation, Source Coding, Mixed Source, Asymp- 
totic Normality 

I. Introduction 

The problem of random number generation is one of the main topics in information theory [I]— [3]. 
There are several problem settings in random number generation. In particular, the resolvability 
problem and the intrinsic randomness problem are representative of them. The resolvability problem 
is formulated as follows [5]. We first use the term of "general source" to denote a sequence X = 
{X™}™ =1 of random variables X n indexed by n (taking values in countably infinite sets), typically, 
n-dimensional random variables. Given an arbitrary general source X = {X n }^ =1 (called the target 
random number) , we generate or approximate it by using a discrete uniform random number whose 
size is requested to be as small as possible. One of the main objectives in this problem is to construct 
an efficient algorithm that transforms the discrete uniform random number to the specified target 
source. Han and Verdii PQ, and Steinberg and Verdii [6] have determined the infima of achievable 
uniform random number rates by using the information spectrum methods. On the other hand, the 
intrinsic randomness problem is formulated as follows in Vembu and Verdii [2J : Given an arbitrary 
general source X = {X n }^ =1 (called the coin source), we try to generate or approximate, by using 
X = {X n }^ =1 , a uniform random number at as large rates as possible. Vembu and Verdii [2J 
and Han [5] have determined the suprema of achievable uniform random number generation rates, 
again by invoking the information spectrum methods. Since the class of general sources is quite 
large and it includes all nonstationary and/or nonergodic sources, their results are very basic and 
quite fundamental. 

Furthermore, it turned out that these random number generation problems have close bearing 
with the fixed- length source coding problem (cf. Han and Verdii pQ). All the formulas established 
here may be said to be ones of the first-order. 

On the other hand, the finer evaluation of achievable rates, called the second-order achievable 
rates, have been investigated in several contexts. In variable-length source coding problem, Kon- 
toyiannis [7j has established the second-order source coding theorem. In channel coding problem, 
Strassen (see, Csiszar and Korner [5]), Hayashi [SJ, and Polyanskiy, Poor and Verdii [TU] have 
determined the second-order capacity rates. Hayashi [11] has also shown the second-order achiev- 
ability theorems for the intrinsic randomness problem as well as for the fixed-length source coding 
problem with general sources. In addition, for i.i.d. sources he has demonstrated the calculation of 
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the second-order optimal achievable rates by using the asymptotic normality in both problems. 

In the present paper, we address the computation problem concerning the second-order for- 
mulas for resolvability, intrinsic randomness, and fixed-length source coding problems for mixed 
(non-i.i.d.) sources, where the resolvability problem was first and partly studied by Nomura and 
Matsushima |12j . In the resolvability problem or the intrinsic randomness problem the degree of 
approximation is measured in terms of variational distance. 

As we have mentioned in the above, the analysis based upon the asymptotic normality is effective 
in deriving the second-order achievable rates. However, it had earlier been applied only to the class 
which has a simple probabilistic structure, such as i.i.d. sources, Markovian sources or stationary 
discrete memoryless channels but not to that of mixed sources. 

In the present paper, we first establish the resolvability formula for general sources, and then 
specifically compute the second-order optimal achievable rates for mixed sources, which is a wider 
but still tractable class of sources than the previous ones in random number generation. Recall 
that mixed sources are typical cases of nonergodic sources. Nonetheless, we show that we can use 
still the two-peak asymptotic normality to compute the second-order achievable rates for mixed 
sources. 

Related works include, e.g., Polyanskiy, Poor and Verdii [13] that has developed the second-order 
capacity of the Gilbert-Elliott channel (GEC). The GEC is a simple model of channels in which 
the crossover probability of a binary symmetric channel obeys a binary symmetric Markov chain 
transition. In particular, they have analyzed the nonergodic case in the GEC using the two-peak 
asymptotic normality from the viewpoint of a mixture of two memoryless channels. On the other 
hand, in this paper we consider a mixed source consisting of two i.i.d sources with countably infinite 
alphabet. Our analysis is on the basis of the information spectrum methods, and hence valid with 
countably infinite alphabet. In addition, it is shown that our results can be easily extended to 
more general cases. Actually, we generalize our results to mixed sources consisting of countably 
infinite i.i.d. sources with countably infinite alphabet, and furthermore to mixed sources consisting 
of countably infinite Markovian sources but with finite alphabet. The resolvability formula for 
mixed sources with general mixture not necessarily countably infinite mixture is also established. 

It should be emphasized here that we have recourse to a bulk of information spectrum calculations 
throughout in the paper. Although they apparently look tedious and even cumbersome, each step 
in the process of computations is actually simple and very basic, which features the information 
spectrum unifying approach. 
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This paper is organized as follows. In Section II, we review the previous results on the first-order 
asymptotics for general sources. In section III, we review and derive the second-order asymptotic 
formulas for the general sources, analogously to Section II. In Section IV, we define the mixed 
source and state the lemmas which play the key role in the subsequent analysis. In Sections V, 
VI and VII, with mixed sources we establish the second-order achievability by invoking the two- 
peak asymptotic normality, for the resolvability problem, the intrinsic randomness problem and the 
fixed-length source coding problem, respectively. In Section VIII, we point out that all the results 
established in Sections V— VII are still valid if we consider more general mixed sources. Finally, we 
conclude our results in Section IX. 



In this section we review the previous results on the first-order asymptotics of random number 
generation and fixed-length source coding. 

To this end, we first give the necessary notations and definitions. In the sequel, let Y = {y'l^j 
be a general source with values in countable sets y n . Let Z be a countable set and let Z, Z be 
random variables with values in Z. Denote by d(Z, Z) the variational distance 



where Px(') denotes the probability distribution of random variable X. Moreover, set Um = 
{1, 2, • • • , M} and let Um denote the random variable uniformly distributed on Um- 

A. First- Order Resolvability 

Definition 2. 1 : Rate R is said to be ^-achievable if there exists a mapping <j) n : Um„ —> y n such 



II. First-Order Asymptotics 



d(Z,Z) = J2\Pz(z)-Pz(z)\ 



that 



lim sup — log Mi 



n 



< R and lim sup (i(Y n , 4> n (U Mn )) < 5. 



Definition 2.2 (5 -resolvability): 



SMY) 



inf {R \ R is ^-achievable } . 



Then, we have 

Theorem 2.1 (Steinberg and Verdu JE/): 




5 



where 




Fig. 2.1. First-Order Resolvability Rate 



B. First-Order Intrinsic Randomness 

Definition 2.3: Rate R is said to be 5- achievable if there exists a mapping <j) n : y n — > Uu n such 
that 

lim inf — log M n > R and lim sup cI(Um , </>„(Y n )) < S. 
Definition 2.4 (5-intrinsic randomness): 

S L (5\Y) = sup{i? \R is (5-achievable } . 

Then, we have 
Theorem 2.2 (Han JSj/J: 



G(R) <^-} (0 < V5 < 2), 



S,(S\Y) = sup ji? 

where 

CZ(m = lim sun Pr i 

n Pyn (Y 



G(R) = lim sup Pr <j - log < R 



The following Fig. 12.21 illustrates Theorem [2T2 



6 




Fig. 2.2. First-Order Intrinsic Randomness Rate 

C. First- Order Fixed-length Source Coding 

Let ip n : y n — > Um u , ipn '■ Uu n y n be an encoder and a decoder, respectively, for source 
Y = {y n }™ =1 . The decoding error probability e n is given by e n = Pr {Y n / ip n (ip n (Y n ))} . Such a 
code is denoted by (n, M n ,e n ). 

Definition 2.5: Rate R is said to be e-achievable if there exists a code (n,M n ,e n ) such that 

limsupe n < e and lim sup — log M n < R. 
Definition 2.6 (e-fixed-length source coding rate): 

Lf(e\Y) = inf {R\R is e-achievable} . 

Then, we have 

Theorem 2.3 (Steinberg and Verdu JE[): 

L f (e\Y) = M{R\F(R) < e} (0 < Ve < 1), 

where F(R) is defined as in (12.11) . 

The following Fig. 12.31 illustrates Theorem 12.31 

An immediate consequence of Theorem 12.11 and Theorem 12.31 is the following theorem, which 
reveals an operational equivalence between resolvability and fixed-length source coding from the 
viewpoint of random number generation, that is, 

Theorem 2.4 (Han JH Remark 2.4-1]): 

L f {e\Y) = S r {2e\Y) (0 < Ve < 1). 
As for the operational meaning of this equivalence, see Appendix |A"1 
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III. Second-Order Asymptotics 

Having reviewed the results on the first-order asymptotics, we now focus on the second-order 
asymptotics, which will turn out to be in nice correspondence with the first-order asymptotics. 



A. Second- Order Resolvability 

Definition 3.1: Rate R is said to be (a, <5)-achievable if there exists a mapping </> n : Um„ — > y r ' 
such that 

lim sup — j= log < R and limsupd(y n , (f> n (UM„)) < 5. 

n— loo v" 6 n— >oo 



Definition 3.2 ((a, 5) -resolvability): 

S r (a, 5\Y) = inf {R \R is (a, <5)-achievable} . 

Then, we obtain the following fundamental formula for the resolvability problem: 
Theorem 3.1: 



S r (a,6\Y) =inf \r 



where 



F a (R) <-} (0 < V5 < 2), 



11 R 
F a (R) = limsupPr <j - log — — — — > a + — 

n ry n {Y n ) ^Jn 



(3.1) 



The following Fig. 13.11 illustrates Theorem 13. 11 

Remark 3.1: This theorem can be driven as a consequence of Theorem 13.31 combined with 
Theorem 13.41 below. See also Remark 13.21 in Subsection IIII-CI below. 




Fig. 2.3. First-Order Fixed-length Source Coding Rate 
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Fig. 3.1. Second-Order Resolvability Rate 



B. Second- Order Intrinsic Randomness 

Definition 3.3: Rate R is said to be (a, <5)-achievable if there exists a mapping </>„ : y r 
such that 

liminf — ^= log — ^ > R and lim sup d(UM ,0n(^ n )) < 5. 



Definition 3.4 ((a, 5) -intrinsic randomness): 

S t (a,5\Y) = sup {R\R is (a, <5)-achievable } . 

Then, we have 

Theorem 3.2 (Hayashi FIT]/): 



5 t (a,5|Y) = supji? 



G (fl) <-} (0 < V5 < 2), 



where 



G a (fl) = hm sup Pr { I log ^-L^ < a + -| 



The following Fig. 13.21 illustrates Theorem 13.21 
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Fig. 3.2. Second-Order Intrinsic Randomness Rate 
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C. Second-Order Fixed-length Source Coding 
Definition 3.5: Rate R is said to be (a, e)-achievable if there exists a code (n,M n ,e n ) such that 

limsupe n < e and lim sup — p= log — — < R. 

n— >oo n— >oo \Tl e 

Definition 3.6 ((a, e)- fixed-length source coding rate): 

Lf(a,e\Y) = ini{R\R is (a, e)-achievable} . 



Then, we have 

Theorem 3.3 (Hayashi f77]/): 



where F a (R) is defined as in (|3.ip . 

The following Fig. 13.31 illustrates Theorem 



L/(a,e|Y) = mf{R\F a (R) < e} (0 < Ve < 1) 




Fig. 3.3. Second-Order Fixed-length Source Coding Rate 



On the other hand, we have the following equivalence theorem between resolvability and fixed- 
length source coding, which is a finer version of Theorem 12.41 for the case of the second-order 
asymptotics: 

Theorem 3.4 (equivalence theorem): 

L f (a, e|Y) = S r {a, 2e|Y) (0 < Ve < 1). 

Proof: This theorem is to reveal a kind of operational relationship between resolvability and 
fixed-length source coding. It suffices to show that if R > Lf(a,e\Y) then ii + 7 > S r (a, 2e|Y) for 
any small 7 > 0, and vice versa with R and R + 7 swapped. To this end, it is sufficient to literally 
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follow the arguments described in Han [5j p. 163] with 7 replaced by -^=. To see its mechanism 
explicitly, we will give the details of the proof in Appendix [A3 ■ 
Remark 3. 2: Theorem 13,11 can also directly, not via Theorem 13.44 be proved by using Lemma 
15.11 and Lemma 15.21 to be described in Section [V] Thus, the second-order resolvability formula 
for general sources can be reasonably established by using the argument similar to that for the 
first-order formula. This is because the lemmas obtained by the information spectrum methods are 
very basic and fundamental. This viewpoint has been pointed out also in Hayashi [TT]. Actually, 
in |llj the second-order formula for intrinsic randomness (Theorem 13. 2\) as well as for fixed- length 
source coding (Theorem I3.3|) has been proved using these lemmas (also cf. Lemmas 17.11 and 17.21 
in Section VII), which had earlier been used already to establish the first-order formulas for these 
problems (cf. Han [5]). 

IV. Second-order asymptotics for mixed sources 

So far we have demonstrated the general formulas for typical first-order and second-order asymp- 
totic problems (resolvability, intrinsic randomness and fixed-length source coding rate) of random 
number generation with any general source Y = {Y n }^ =1 . 

However, computation of these general formulas is quite hard in general or even formidable. 
Therefore, in this section we consider to introduce a class of tractable sources Y for which the 
general formulas are computable but still of independent interest. One of such source classes would 
be the case where Y is a mixed source of two i.i.d. sources Yi = {Y 1 n }^_ 1 and Y2 = {Y^nLi- The 
computation problem of the first-order asymptotics for such mixed sources has already been solved 
(e.g., see Han [5]), so in the sequel we now focus on the computation problem of the second-order 
asymptotics for mixed sources. As a result, it will turn out that we can explicitly compute the 
asymptotic formulas by virtue of the information spectrum methods. 

Let us begin with the formal definition of mixed sources. Let y = {0, 1, 2, • • • } (countably infinite) 
be a source alphabet and y = yiy2 • • • Vn £ y n denote a sequence emitted from the source of length 
n. Let Y n denote a random variable: a source sequence of length n. 

We consider a mixed source consists of two stationary memoryless sources Yj = {Y™}^ =1 with 
2 = 1,2. Then, the mixed source Y = {Y n }^ =1 is defined by 

iV»(y) = w(l)Py~(y) + w(2)Pyn(y), (4.1) 

where w(i) are constants satisfying w(l) + w(2) = 1 and w(i) > (i = 1, 2). Since two i.i.d. sources 
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Yj (i = 1, 2) are completely specified by giving just the first component Yi {i = 1, 2), we may write 
simply as Yj = {Y} (i = 1, 2) and define the variances: 
Definition 4-1 (variance): 



1 



af = E { log PM)- Hm ) (i = 1,2) 

where we assume that these variances are finite, and define the entropy by 

Since we consider the case where Yj = {Yi] [i = 1,2) is an i.i.d. source, the following asymptotic 
normality holds for each component i.i.d. source: 



logP Y ™(Y t n )-nH(Yi) ^ tt ] f +co 1 



lim Pr <^ ' v ' / — > U } = / -= exp 

[ V ncr i J JU V2vr 

where ai 2 denotes the variances defined in Definition 14.11 (i = 1, 2). 



z 



dz, (4.2) 



The following lemma plays the key role in dealing with mixed sources in the proof of Theorem 
5-H Theorem 16.11 and Theorem 17.11 In other words, the crux of the arguments for mixed sources 
in the present paper is summarized by Lemma l4.lt which is completely irrelevant to the matters 
posed by Hayashi |llj . 

Lemma 4-1 (Han J^): Let {z n }^ = i be any real- valued sequence. Then for the mixed source Y 
it holds that, for i = 1,2, 



n vn 



Pr ( - '° g ^ > 4 < Pr { - '° S % > ^ - 7 „ > , 



where 7„ > satisfies 71 > 72 > • • • > 0, 7„ — > 0, \/nj n — > 00. 
Proof: See Appendix |Bj ■ 



V. (a, <5)-RESOLVABILITY 

In this section we shall establish S r (a,6\Y) for mixed sources. At first we introduce here two 
fundamental lemmas of Han [5]: Lemma 15. II and Lemma 15.21 below. Before describing the lemmas, 
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we need to define two sets. Let X = {J n }™ =1 and Y = {Y n }^ =1 be arbitrary general sources with 
countably infinite alphabets, and given a sequence {z n }^ =1 define S n (z n ) and T n (z n ): 



S n (z n ) = <jx G X n 
T n (z n ) = fyef 



1 , 1 

(y) 



Lemma 5.1: Let X = {X n }^ =1 and Y = {y n }^ =1 be arbitrary general sources, where X n and 
are random variables taking values in X n and y n , respectively. Then, for an arbitrary sequence 
{ z n}%Li and 7 > 0, there exists a mapping (fr n : X n — > y n such that 

d(MX n ),Y n ) < 2max(Pr{X™ £ 5 n (z n + 7 )}, ??{Y n £ T n (z n )}) + 2e~^. 

Lemma 5.2: Let X = {X n }^ =1 and Y = {Y n }^ =1 be arbitrary general sources, where X n and 
Y n are random variables taking values in X n and y n , respectively. Then, for an arbitrary sequence 
{zn}n=n 7 > and any mapping <p n : X n -> y n it holds that 

d(0„(X"),y") > 2Pr{Y n i T n {z n + 1 )} - 2Y>v{X n G 5 n (^)} - 2e"^. ( 5 .1) 

Remark 5.1: Also, (15. ip can be written as 

d(MX n ),Y n ) > 2Pr{X n i S n {z n )} - 2Pr{Y n G T n (z n + j)} - 2e~^ . (5.2) 

The above lemmas are useful for the random number generation problem to approximate a 
probability distribution Y = {Y n }^ =1 by using an another probability distribution X = {X n }^ =1 . 
Clearly, this includes the resolvability problem as a special case, which is the case of X n = Um„ ■ 
Therefore, in this case the condition in the above lemmas leads to 

{0 z n < 4= log M n 
~ ^ (5.3) 
1 z„>^logM n . 

Notice that the above lemmas are valid for general sources X and Y. 

In the sequel, we consider the case that < 5 < 2 and w(l) / f hold (cf. Remark O below 

for the case of w{l) = § ). Then, given < S < 2 we classify the problem into three cases. Here, 

without loss of generality, we assume that H{Y\) > H(Y 2 ) holds: 

I H{Yx) = H(Y 2 ) holds. 

II H(Yl) > H(Y 2 ) and w(l) > f hold. 

III H{Y 1 ) > H(Y 2 ) and w(l) < § hold. 
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In Case I, we shall establish S r (H(Yi),S\Y) (obviously, this is equal to S r (H(Y2),5\Y)). In Case 
II and Case III we shall show S r (H(Yi), 8\ Y) and S r (H(Y2), S\Y), respectively. Now we have one 
of the main results: 

Theorem 5.1: Given < 5 < 2, the following holds. 
Case I: 



S r (H(Y 1 ),6\Y)=T 1 



(5.4) 



where T\ is specified by 



Case II: 



,oo 1 


- z 2 " 


/ — 7= exp 






2 



dz. 



(5.5) 



S r (H(Y 1 ),6\Y)=T 2 , 



(5.6) 



where T 2 is specified by 



,oc l 


' z 2 " 


/ ^=exp 






2 



dz. 



(5.7) 



Case III: 



S r (H(Y 2 ),S\Y)=T 3 



(5i 



where T3 is specified by 



w(l) + w(2) 



f°° 1 


" z 2 " 


/ — 7= exp 






2 _ 



(5.9) 



Illustrative figures in each case of this theorem are depicted in Fig. l5.1| -Fig. l5,3l where the weighted 
probability of the shaded area is equal to |. 
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Remark 5.2: It is easy to check that Ti = — oo, T3 = +00 for w(l) = |. Also, it will turn 
out from the way of proving the above theorem that the second-order asymptotics gets trivial if 
a ^ H(Y\) and a 7^ H(Y2), because this case necessarily implies that 5 = or 5 = 2w(l) or 5 = 2, 
depending on the value of a; then, accordingly, we can formally set as S r (a,5\Y) = —00. 

Remark 5.3: Theorem 15.11 can be restated more intuitively but equivalently as follows. Let 

R n = -\ogM n 
n 

denote the size rate of resolvability (cf. Definition 13.11 and 13 . 2[) , and consider the following asymp- 
totic equation for R n : 



w 



where $(•) is the Gaussian cumulative distribution function defined by 



/2tt 

Denote the solution of this equation by 



-00 



<&(x) = / e 2 dx. 



and set as 



W n = -\ogM* 
n 



R* = a -\ — ^= + o \—= I (a, b are constants), 
Jn \\ n ' 



which, substituted into (I5.10p . yields 

Y^ + b + o 

V\ ) ' ' V °2 J 2 



1 - + b + oi - 1 A + ^ f - »w) + " ± °w) = £ . (5 . n) 




Fig. 5.1. Case I 
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Fig. 5.2. Case II 




Fig. 5.3. Case III 

Then, it is not difficult to verify by letting n — > oo that, given a and 5, the corresponding solution 
b = b*(a, <5) of this equation coincides with T\, T2 and T3, respectively, according to Cases I, II, III 
(cf. also (|8.3|) in the proof of Theorem 18. ip . Notice here that the equation (|5.10p subsumes Remark 
l5.2l too. Thus, it is concluded that b* (a, 5) is nothing but the second-order resolvability S r (a, <5|Y), 
and hence Theorem 15.11 is equivalent to the equation (|5.10p . 
Summarizing up, we can write the optimal size M* as 

log M* = na + \/nb* (a, 5) + o(\/n), 
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with a = H(Yi) or a = H(Y 2 ) depending on the value of 6, that is, if H{Y\) ^ H(Y 2 ) then 
a = H(Y l ) ^b*(a,8)=o 1 $- l (^ m ) if w{\) > § ; 
a = £T(y 2 ) and 6*(a, 5) = o 2 $>~ 1 (^j 1 ) if «;(1) < f , 
which enables us to evaluate how large size of M* is needed as a function of block length n and 
variational distance 5. Notice here that in this case b* (a, 5) can be written as the simple inverse 
function <5 _1 of the Gaussian distribution function, and also that b* (a, 5) can be negative, for 
example, b*(a, 5) < if 5 > 1 + w(l), so that in this case the first-order <5-resolvability S r (<5|Y) is 
H(Y2) but the optimal achievable rate i?* approaches it from below. In other words, it is possible 
to make necessary rates to be below the <5-resolvability at finite block length n. The nonergodic 
channel counterpart of the equations f|5. 1Q[) and (I5.12p has been provided by Polyanskiy, Poor and 
Verdii |13j . who have observed for the Gilbert Elliott channel the same kind of non-asymptotic 
phenomenon as here. On the other hand, in the case where H(Y\) = H(Y2) holds, b* (a, 5) can be 
written as the inverse function of a mixed Gaussian distribution function (see, Case I). Notice here 
that Case I is missing in |13j . ■ 

Proof of Theorem \5. 1[ See Appendix [Cl ■ 

VI. (a, <5)-INTRINSIC RANDOMNESS 

Let us now turn to the computation problem of the (a, <5)-intrinsic randomness formula for mixed 
sources. To do so, without loss of generality, we consider the following three cases: 

I h(Yl) = H(Y 2 ) holds. 

II H(Y 1 ) > H(Y 2 ) and w(2) > f hold. 

III H(Yi) > H(Y 2 ) and w{2) < | hold. 

Then, we have 

Theorem 6.1: Given < \/8 < 2, the following holds. 
Case I: 

S L (H(Y 2 ),S\Y)=n, 

where T4 is specified by 
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Case II: 



where T5 is specified by 



Case III: 



S L (H(Y 2 ),S\Y)=T 5 



- = w(2) 
2 v ' 



T 5 
j "2 


1 


" z 2 - 




-== exp 




1 —CO 


v/2tt 


2 



dz. 



S L (H(Y 1 ),6\Y)=T 6 , 



where Tq is specified by 



- = w (2)+w(l) 



1 



exp 



v2H 



dz. 



-00 V27T 

Proof: It suffices to proceed in parallel with the arguments as made in the proof of Theorem 
15. 1| while taking account of the duality between resolvability and intrinsic randomness. Note that 
(|5.2p is used instead of (|5.ip in the proof of Converse Part. ■ 
Remark 6.1: It is easy to see that T5 = +00 and Tq = —00 for w(2) = |. The latter part of 
Remark 15.21 similarly applies here too with S L (a,5\Y) = +00 instead of S r (a,5\Y) = —00. Also, 
like in Remark 15. 3} we can ascertain that Theorem 16.11 is equivalent to the asymptotic equation: 

ro(1) * (Mil) + w{m (v^WlzSM) = |, 

where ty(x) = 1 — and T4, T5, Tg instead of T±, T2, T3, respectively. 

VII. (a, e)-FIXED-LENGTH SOURCE CODING 

Let us now consider to compute the formula for Lt(a,e\Y) for mixed sources. To do so, without 
loss of generality, we consider the following three cases: 

I H{Yx) = H(Y 2 ) holds. 

II H(Yi) > H(Y 2 ) and w(l) > e hold. 

III H(Y l ) > H(Y 2 ) and w(l) < e hold. 
Then, we have the following main result: 

Theorem 7.1: Given < e < 1, the following holds. 
Case I: 



L f (H(Y 1 ),e\Y)=T 7 , 



(7.1) 
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where Tj is specified by 



i=l 



,00 1 


" z 2 " 


/ ^=exp 




Jzl x/2tt 


2 _ 



Case II: 



where Tg is specified by 



Case III: 



where Tg is specified by 



L f (H(Y 1 ),e\Y)=T 8 , 



(7.2) 



(7.3) 



e = w(l) 



,00 1 


- z 2 " 


/ ^=exp 






2 



L/^yz), e|Y)=r 9 , 



tu(l)+ie(2) 





" z 2 " 


/ — 7= exp 




/2k \/2¥ 


2 



Remark 7.1: It is easy to check that Tg = —00, Tg = +00 for w(l) = e. The latter part of Remark 



[52] similarly applies here too with e instead of | and Lf(a, e|Y) = —00 instead of S r (a, <5|Y 

S 
2 



-00. 



Remark 15.31 applies also here with e instead of |, and T7, Tg, Tg instead of T\, T2, T3, respectively. 



In particular, it turns out that Theorem 17.11 is equivalent to the equation (|5.10p . 

Proof of Theorem \ 7. 1\ Although the proof is immediate from Theorem 13.41 and Theorem 15.11 



with e = I , we give in Appendix [D] another information spectrum approach that is of independent 
interest, where Lemma 17. II and Lemma 17.21 (due to Han [5]) as described below are invoked, which 
Hayashi |llj has fully used to obtain the general formula for the second-order optimal fixed-length 
source coding rate (see, Theorem I3.3P but not for mixed sources. ■ 
Lemma 7.1: Let M n be an arbitrary given positive integer. Then, for all n = 1, 2, • • ■ , there 
exists an (n,M n ,e n ) code such that 



£n < Pr 



1 



log 



1 



> 



togiWr, 



yfii to Py„(y«) " ^ 

Lemma 7.2: For all n = 1, 2, • • • , any (n, M n , e n ) code satisfies 



e n > Pr 



1 



log 



1 



yfii Py™ (Y r 

where 7 > is an arbitrary constant. 



> -^logM n + 7 }>-e 
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VIII. Second-Order Resolvability for mixed Sources: General cases 

In the previous sections, we have established the second-order achievable rates for a mixture of 
two i.i.d. sources, which is regarded as being the simplest model of mixed sources. In this section, we 
shall extend our results to more general mixed sources. Although we consider only the resolvability 
problem in this section, similar extensions can be done immediately for the intrinsic randomness 
problem and the fixed-length coding problem. 

A. (a, 5) -Resolvability for a Mixture of Countably Infinite i.i.d. Sources 

Let us now consider the mixed source consisting of countably infinite stationary memoryless 
sources Yj = {Y^ c ^' =l , where iGZ = {0, ±1, ±2, • • • }. The mixed source that we consider in this 
subsection is defined by 

oo 

iV-(y)= £ w(i)Prn(y), 



l = — oo 



where w(i) are constants such that ^2 < ^ = _ OQ w(i) = 1 and w(i) > for all i G Z. The variance 
of (i £ Z) of the stationary memoryless source Yj is defined in a similar way to Definition 14. II and 
we assume that all these variances are finite. Finally, without loss of generality we assume that 

■ ■ ■ < (Y_ 2 ) < H(Y_i) < H(Y ) < (Fx) < H(Y 2 ) < ■ ■ ■ . 

With this definition of mixed sources, we now have the following second-order resolvability theorem, 
which is a substantial generalization of Theorem 15.11 

Theorem 8.1: For a mixture Y of countably infinite i.i.d. sources with countably infinite alphabet, 
the optimal size rate .R* = — log M* is given as the solution for R n of the asymptotic equation: 



n 

oo 



Z«0*( W-- H ™ )- l t , (8.1) 



and, furthermore, the resolvability S r (a, S\Y) is given as the solution b = b*(a, 5) of the asymptotic 
equation 



yj wm (M-m)) + _ t (8 . 2) 



Proof: The derivation of equation f)8. 1|) and (j8.2|) is based on the multi-peak asymptotic 
normality instead of the two-peak asymptotic normality. Although it is easy to verify that Theorem 
18.11 follows in a manner similar to that in Remark 15. 3( for the sake of reader's convenience we 
demonstrate here also the formula of the form as in Theorem 15.11 Define the sets of indices as 



£ (a) = {k e Z\H(Y k ) = a, w{k) > 0}. 
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Ci(a) = {i£ Z\H(Yi) > a, w{i) > 0}, 



We first consider the case where Co(a) = (the empty set). This case necessarily implies 

Y w ( i ) = ^ 

iECi(a) 

which allows us to formally set as S r (a, S\Y) = — oo. 

We next consider the case of Cq{o) ^ 0. It is easy to see that letting n — > oo in (|8.2p yields the 
following non- asymptotic equation to determine the second-order resolvability b = b*(a,5): 

Y «;(.•)* (£)+ E "0') = f ( 8 - 3 ) 

In order to specifically compute the b = b*(a, 5), we need several classifications as in the proof of 
Theorem 15. H where equation (|8.3p implies that a given 5 is specified by the following conditions: 

-oo); If C (a) / then 
6 



If £ o (a) = 0then£ j6 A(a)H?) = ! 



Y w U) 



< 



jed(a) 



2' 



Y + Y w W - o- 



This is illustrated in Fig. 18.1 




Fig. 8.1. Countably Infinite i.i.d. Sources 



It is easy to see that these conditions yield Cases I, II, III in Theorem 15.11 as a special case. ■ 
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B. (a, 5)- Res olv ability for a Mixture of Countably Infinite Markovian Sources 

Next, let us consider a mixed source consisting of countably infinite Markovian sources with 
finite source alphabet Yj = {^""j^Li, where i G Z = {0, ±1, ±2, ■ ■ ■ }. The mixed source that we 
consider in this subsection is defined by 

oo 

iV-(y)= E w (^-(y). 

i=— oo 

where are constants satisfying X]£-oo = 1 an< ^ 10 00 — f° r ah * S Z and each Yj = 
{1^™}^]. denotes the stationary ergodic Markovian source with irreducible transition probability 
Qi(k\j) ((j)^) ^ y 2 )> where j and denote consecutive symbols in the Markov transition. We let 
7Tj(-) denote the stationary distribution of Yj. The probability of a sequence y = y±y2 ■ ■ ■ y n € y n 
emitted from Markovian source Yj is given by 

n-l 

Pyn(y) = TTi(yi) J| Qi(3/j+i|2/j)- 

The entropy rate H{Qi) (i S Z) and the variance af (i E Z) of Markovian source Yj are defined 
by 



(fcj)ey 2 v 1 ' 



and 



E ^^(^(log^TuT-^ 

+ 2 yj *tfXWi|*XM*W (log ^ - *(«,)) (log -Ljj - m 



respectively (see Hayashi VII]). In addition, we assume that all these variances are finite, and 
it is also assumed that 

• • • < F(Q_ 2 ) < ff(Q-i) < fl-(Qo) < ff(Qi) < fl"(Q 2 ) < • • • 

With this definition of mixed sources, we now have the following second-order resolvability 
theorem: 

Theorem 8.2: For a mixture Y of countably infinite Markovian sources Yj (i £ Z) with finite 
source alphabet, the optimal size rate i?* = MogM* is given as the solution of the asymptotic 
equation: 



f; W-g(ft)A = i. 
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and, furthermore, the second-order resolvability S r (a,5\Y) is given as the solution b = b*(a,5) of 
the equation 

y/E(a-H(Qi)) + b + o(l) \ = 6 
ai ) 2' 

Proof: Since we assume that the source alphabet y is finite, the number of states of this 
Markov chain is finite. Then, the asymptotic normality also holds due to the central limit theorem 
for Markov chains (see, e.g., Chung [14], Billingsley |15j). Therefore, it suffices to proceed in parallel 
with the arguments as made in the proof of Theorem 18.11 and so we omit the details. ■ 
Remark 8.1: As shown in this section, our analysis for mixed sources is on the basis of the 
asymptotic normality of self-information. This means that the similar argument is valid for any 
mixture of count ably infinite sources for which the asymptotic normality of self-information holds 
for each of the component sources. Generally speaking, one of the conditions to ensure the central 
limit theorem for weakly dependent random variables is, e.g., the mixing condition (see, Billingsley 
PS], Shields 

C. (a, 5) -Resolvability for Sources with General Mixture 

In this subsection we consider a possible extension of Theorems 18.11 and 18.21 to the case with 
general mixture instead of countably infinite mixture. It suffices here only to focus on the extension 
of Theorem 18. 11 because the basic logics underlying both extensions are the same. A mixed source 
Y = {Y n }^ =l with general mixture is defined by 

iV-(y)= / Py?<y)dw{0% (8.4) 

J A 

where w{9) is an arbitrary probability measure on the parameter space A, and Yg = {Y' (? n }^ =1 (Q 6 

A) are i.i.d. sources with finite alphabet. 

With this definition, we have the following formal extension of Theorem 18.11 

Theorem 8.3: For a source Y with general mixture w(0) of i.i.d. sources Yg with finite alphabet, 

the optimal size rate R* n = — log M* is given as the solution for R n of the asymptotic equation: 

! • (&&z*sm dw{s) = i, 

J A V a 6 ) 2 

and, furthermore, the second-order resolvability S r (a,5\Y) is given as the solution b = b*(a,S) of 
the asymptotic equation 

t , r yt.(a- mm +b+ om\ _ , 

J A V u & J 2 



«>(*)* 
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where is the variance of log Py . 

Proof: The second inequality of Lemma 14.11 is not necessarily valid in the case with general 
mixture. Nevertheless, we can slightly modify it so as to be applicable to this general case. As for 
the details, see the proof of Han [51 Lemma 1.4.4]. ■ 
Remark 8.2: Let us define, instead of Cq{o) and £\{a) as above, the subsets Ao(a) and Ai(a) of 
A as follows: 

Ao(o) = {9 e A\H(Y e ) = a}, 

Ax (a) = {9 G A\H(Y e ) > a}. 

Then, the equation (|8.3|) to determine the (a, 5)-resolvability b = b* (a, 5) is extended accordingly 
to the case with general mixture w(9) as follows: 



f $ (—) dw(9) + / 

JA (a) K^eJ J A, 



dw{9) = S -, (8.5) 

'Ao(a) \°~o; JA 1 (a) 1 

which implies that a given 5 is specified by the following conditions: If J*A (a) dw(9) = then 
/a i( «) MO) = l(b = -oo); If f Ao(a) dw{9) > then 

dw(9) < S -, 

Ai(a) 2 

[ dw{9) + / dw{9) > -. 

JAo(a) JA 1 {a) 1 

IX. Concluding Remarks 

We have so far considered the second-order achievability to evaluate the finer structure of random 
number generation for mixed sources. The class of mixed sources is very important, because all of 
stationary sources can be regarded as forming mixed sources obtained by mixing stationary ergodic 
sources with respect to appropriate probability measures. Although, in general, mixed sources do 
not have the asymptotic normality. So, our result is also meaningful, we have demonstrated that 
the analysis based on the two-peak asymptotic normality is still effective also for sources whose 
self-information spectrum does not have a single asymptotic normality. 

As shown in the proofs of the present paper, the information spectrum approach is substantial in 
the analysis of the second-order achievable rates. In particular, Lemma 14. II is a simple but enables 
us to work with the multi-peak asymptotic normality for mixed sources (cf. the proof of Theorem 
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Polyanskiy, Poor and Verdu |13j has derived the second- order capacity in either ergodic or noner- 
godic setting for the Gilbert-Elliott channel (GEC), which consists of two binary symmetric chan- 
nels. Their results for the nonergodic case is also based on a mixture of two Gaussian distributions. 
On the other hand, since our analysis is based on the information spectrum methods, the results 
are valid also for a mixture of countably infinite sources (i.i.d. with countably infinite alphabet or 
Markov with finite alphabet) as well as a general mixture of i.i.d. sources with finite alphabet as was 
shown in Section VIII. It should be emphasized that, throughout in the paper, we have established 
the asymptotic or nonasymptotic equations for determining the second-order resolvability, intrinsic 
randomness, and fixed- length source coding rate. The forms of these equations include that as 
shown in |13j as a special case. We observe that the second-order capacity of finite state Markov 
channels (see, [T7]) or Fritchman channels (see, |18j). which is generalizations of the GEC, can be 
established by means of the similar arguments. This will be reported in a forthcoming paper |19j . 



This appendix concerns the operational relationship between resolvability and fixed-length source 
coding. Although the proof literally mimics the argument given by Han [5J p. 163], we will repeat 
it here for the reader's convenience. 



First, W6 sliow how to construct cin (n,, A/?^, £ n ) source code, given a resolvability mapping <f>. 
Um u y n - Set Y n = <p n (UM n ), then we can define the subset So of y n by 



Clearly, we have |<So| < M n which enables us to define the source encoder ip n : y n — > Um„ which 
transforms each element of So to a distinct element of Um„ and all elements of y n \Sq to 1. In 
addition, if we define the source decoder ip n as the inverse mapping of (p n \s a -, the (ip n , tp n ) becomes 
an (n, M n ,e n ) source code for the source Y = {Y n }^ =1 such that 



Appendix A 



Proof of Theorem I3.4I 



So = {y e y n \Py n (y) > 0} . 



e n = Pr{Y n <£S }. 



Here, from the other definition of the variational distance: 



d(Z, Z) = 2 sup \P Z {A) - Pz(A) 



A:AcZ 



it follows that 
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Considering that Pr |y n G 5q| = 1 holds, we see that 



Pr{y" ^S }<\d(Y n ,Y n ). 

Consequently, we obtain 

e n <±d(Y n ,Y n ). (A.l) 

In this way we can construct an (n, M n ,e n ) source code satisfying (lA.lj) with the same M n as in the 
resolvability mapping <j) n : Um„ — > y n , which implies that if R > S r (a, S\Y) then R > Lf(a, ||Y). 

Next, let us show how to construct a resolvability mapping <p n , given an (n, M n ,e n ) source code. 
We can define the subset So of y n by 

S = {y e y n \y = MVn(y))} , 

where (fn^n) denotes the pair of encoder and decoder of the (n, M n ,e n ) code. For an arbitrarily 
small 7 > set M' n = M n e v/ "' r '. From the same argument as in the proof of Lemma 15.11 (cf. Han 
[5]) with Um^, <So and Um'„ m place of X n , T n (z n ) and S n {z n + 7), we can construct a mapping 
4>n ■ Uw n y n such that 

d(Y n ,Y n ) < 2Pr{y n i S } + 2e~^, (A.2) 

where Y n = <^ n (C^M;)- Since e n = Pr{y n ^ So}, (IA.2p can be expressed as 

d(Y n ,Y n ) < 2e n + 2e~^. 

In this way we can construct the resolvability mapping Y n = (f> n (UM^), which implies that if 
R > L f (a, e|Y) then R + j> S r (a, 2e|Y). 

Appendix B 
Proof of Lemma 14.11 

Although the proof to be shown below is implicitly contained in Han [5], we explicitly summarize 
it here for the reader's convenience. At first we show the first inequality. Set a sequence {7 n }^ c L 1 
satisfying 71 > 72 > • • • > 0, 7„ — > 0, \A^7n ~~ * 00 • Then it holds that 

f -lOgiVn(^) _ -bg iVH*?) 

1 / — / — — 



< J2 ^V"(y)e _>/ " 7 " < e~^ 7 " 



yer>„(») 
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for i = 1,2, where 



D n (i) = <yey 



-iog/v»(y) -iogiV(y) 



n 



< ~7ri 



This means that 



holds for i = 1, 2. So we have 



> 1 - e _ ^». 



-iog^(y^) -i og iv»(i?) 

7n < 



/I 



with probability 1 — e v**7». g 0) we have for i = 1, 2 



7n > Z n 



which is the first inequality of the lemma. Secondly, we show the second inequality of the lemma. 
Set 

S n {z n ) = |y £ y n |iVn(y) < e"^" } 



iV 4 »(y) < 



w(i) 



From the property of mixed sources, y € Sn\z n ) (i = 1, 2) holds for y G S n {z n ). This means that 

(i = 1, 2). Moreover, since, by assumption, 7„ > — '"s^^) ^ _ ^ 2) hold for sufficiently large n, we 
have 



n 

= Pr{Py n (y") < e-^"} 

= P Kl "(S„(Zn)) 

Tog^lT) -Iogw(i) 



■;?. 



40g_^n(l7) 

< Pr<J ^ >^n-7n^, 



■;?. 



for sufficiently large n, which is the second inequality. 
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Appendix C 
Proof of Theorem 15.11 

Proof of Case I: 

A simplest way to prove Theorem 15.11 is to first apply Theorem 13.11 to the present case of mixed 
sources and to proceed to the computation of necessary quantities. Here, however, more basically 
we start along with Lemma [5. II and Lemma [5. 2 1 in order to reveal the fundamental logic underlying 
the whole process of random number generation. Actually, the computations needed in the first way 
of proof are contained in those needed in the second way of proof; more exactly, both computations 
are substantially the same. 

We show the proof of Case I by using Lemma 15.11 and Lemma 15.21 The proof consists of two 
parts. 



1 ) Direct Part: 

Set M n = e n #( y i)+ T iv / "+7 5 w here 7 > is an arbitrarily small number. Then, trivially it holds 
that 

lim sup — < 1 1 . 

n— >-oo \/n 



Thus, it is enough to show that there exists a mapping <j) n such that 

limsu pC ^ n ([/ M J,y n ) < 6. 

n— >oo 

On the other hand, set z n = nH ( Yi )^_ Ti Vn _ ^ then z n , + 7 < loj ^" holds. Thus, from Lemma 
15.11 and (|5.3p there exists a mapping <p n such that 

( nH(Yi) + T\Jn 1 1 ] 

Moreover, from Lemma 14.11 there exists a mapping <f> n such that 

limsup^(0 n (C7 M J,^ n ) 

f nH(Y 1 )+T 1 ^i 1 1 

< hm sup rr < — 7 < —= log 



fn 1 y/n & P Y ™(Y* 



n—>-oo ^ 

A (nH(Y 1 )+T 1 ^E 1 1 

hmsupgPrj - 7<^lo g7R ^y> 
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2 f 

< lim sup Pr < 

«=1 v 



nHiY^ + T^ 1 1 
7 - 7n < -f= log ■ 



w{i) 



< > lim sup Fr^ 5 ^= > — 27 f u>(z), 



i=l 



I? 



because j n > is specified in Lemma l4.lt an d so 7„ < 7 holds for sufficiently large n. Then, noting 
that H(Yi) = HiY?) holds, we have from the asymptotic normality (by virtue of the central limit 
theorem) 

logPY^YD-nHjY^ T X - 27 
lim sup Fr { — > 



n— >oo 

POO 



1 



00 



cxp 



noi 
2- 



1 



cxp 



dz+ 



exp 



Here, by the continuity of the normal distribution, 



T i- 2 ~< \/2tt 



cxp 



->■ 



as 7 0. Thus, noting that 7 > is an arbitrarily small, we have 

1 



limsup -d((t> n {U M J,Y n ) 



n— >oo 
2 



roc 1 


" z 2 " 


/ -7= exp 






2 _ 



Therefore, Direct Part has been proved. 



2) Converse Part: 

We consider a constant T[ < 7i. Assuming that T[ is (-ff(li), (5)-achievable, we shall show a 
contradiction. Since we assume that T[ is (H(Yi), <5)-achievable, there exists a mapping <^> n such 
that 

limsupd( ( / )n (C/ M J,y n ) < 5, 

v logMn-nHjYj) _ 
lim sup — < 1 1 , 



n 



which means that there exists a constant 7 > satisfying 

log M n - nH(Yi 



n 



< T{ + 7 < Ti 



for sufficiently large n. 
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On the other hand, set z n = nH ( Yi y±_ T iV™ _|_ ^ Then z n > lo ^f" holds. Thus, from Lemma 15.21 
and (|5.3p . for any mapping <p n it holds that 

i 



-d(MU M J,Y n ) 



> Pr 



1 



Thus, from Lemma l4.lt f° r an Y mapping n we have 
liminf^(0 n (C/ M J,y n ) 



-Vn7 



> liminf Pr 



(Y l)+ r;vn +27 < 1 1 



: lim inf > Pr 



i=l 



nHiYA+TiJ^ 1 1 

v_ +27< _ log __ h 



> 



^ lim inf Pr V U lV + 2 7 + 7n < — log — -— 

2 

Elim inf Pr 
n. — ^Dn 



101 



i=l 
2 



\ogP Y n{Y l n )-nH{Y 1 



> 



Elim inf Pr 



i=l 



logP vr (Yl l )-nH(Y 1 ) 



n 



> T{ + 3 7 \ w{i) 



(C.l) 



because 7 > 7n holds for sufficiently large n (cf. Lemma l4.ip . Then, by virtue of the asymptotic 
normality, for i = 1, 2 it holds that 



lim inf Pr 

n— ¥00 



logiV 4 »(*?) - n ff(yi) T{ + 3 7 



1 



> 



00 



exp 



" 2 



exp 
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dz. 



'a V2vr 

because we can let T{ + 3 7 < T\ hold by letting 7 — )• 0, and by substituting the above inequality 
into ()C.ip for any mapping n , we have 



liminf -d{<j, n (U Mn ),Y T 

n— >-oo z 



i=l 



,00 J 


- z 2 - 


/ — 7= exp 






2 _ 



dz 



This is a contradiction. Therefore, the converse part has been proved. 



Proofs of Case II and Case III: 
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The proofs of Case II and Case III are similar to the proof of Case I. Here, we show only 
differences in the proofs. 



1 ) Direct Part: 

a) Similarly to Case I: we have the proof of Case II as follows. 
limsup^ n ([/M„),^ n ) 



n— >oo 
2 

<" 



> hm sup Pr <j ■ — > T 2 — 2 7 f w(i) 



i=i 
2 



> rim sup Pr — j=— > — f w{i) 



i=l 



y/n(Ti 



-\ogP Y ^)-nH{Y x ) r 2 -2 7 \ 
lim sup rr < 1= > r w( 1 ) 



°i 



\ogP Y ~{Y 2 n )-nH{Y 2 ) n(H(Y 1 )-H(Y 2 )) T 2 -2 1 \ 
+ hmsupPr \ — y=— > y=- 1 f w{2). 



v / ncr 2 \Jno 2 a 2 

Here, note that H(Y\) > H(Y 2 ) holds. This means that for any constant W\ > 

n{H(Y 1 )-H(Y 2 )) , T 2 -2 7 



+ 



> Wi, 



\[no 2 a 2 

holds for sufficiently large n. Thus, taking account of H(Yi) > H(Y 2 ), we have 

dog P Y ~(Y 2 n )-nH(Y 2 ) njHjY^-HjY,)) T 2 - 2 7 
hm sup Pr <; — p= > h 



y/na 2 

-\ogP Y ,(Y 2 n )-nH(Y 2 ) 

< hm sup Pr < — — > W\ 

\Jno 2 

dy. 



\Jna 2 



a 2 



f°° 1 


' y 2 ' 


/ ^=exp 






2 _ 



Since W\ > can be arbitrarily large, we have 

Hm p f -logiV~(l?)-ng(y 2 ) > n(H(Y 1 )-H(Y 2 )) + T 2 - 2 7 
n^oo I y/na 2 \/na 2 a 2 

Thus, from the asymptotic normality it follows that 

2 nm sup Pr < — > 1 2 — 27 > w{i) 

. n— too V v n J 

1 = 1 v v ; 

f -logPY^Y^-nHjY,) T 2 -2 7 \ 
■ lim sup Pr 1 — — > f w(l) 



0. 



=«;(!) 



exp 



z 2 " 



0*1 
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>w{l) 





- z 2 - 


/ ^=exp 




Jl2 


2 



dz= 5 -, 



by letting 7—7-0. Thus, we have proved the direct part of Case II. 



b) In Case III: we have 

limsup^ n ([/ A/ J,r n ) 

< y nm sup Pr < — > J3 — 27 > w[i) 

1=1 v v y 

dogP^(Y 2 ")-nff(y 2 ) ^ r 3 -27 \ 

< w(l) + limsuprr < — — > } w(2) 



\fn02 



<?2 



w(l)+w(2) 



poo 


1 


" z 2 - 




-j= exp 




lis. 

CT 2 


v/2tt 


_ 2 



dz 



by letting 7—^0. Consequently, similarly to the proof of Case I or Case II, we can prove the direct 
part of Case III. 



2) Converse Part: 

a) In Case II: we consider a constant T' 2 < Ti and we assume that T' 2 is (H(Yi), <5)-achievable. 
Notice that there exists a constant 7 > satisfying T' 2 + 7 < Ti. Then, similarly to the proof of 
Case I, we have 

liminf^ n ([/ M J,y n ) 

> y limint rr< — > i 2 + ^7+7n } w(i) 

i=l k 
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because we can let T 2 + 37 < T2 by letting 7 — >• 0. Hence, we have shown the converse part by 
using the similar argument to Case I. 

b) Similarly, in Case III: we consider a constant T3 < T3 and we assume that T3 is (H(Yi),5)- 
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achievable. Notice that there exists a constant 7 > satisfying T3 + 7 < T3. Then, we have 
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Then, the first term of the right-hand side of the above inequality can be determined as follows. 
Notice that H{Y\) > H(Y2) holds. This means that for any constant W\ > 

n(H(Y 2 )-H(Y 1 )) , T^ + 3 7 



+ 



<7l 



< -Wi, 



holds for sufficiently large n. Thus, 
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holds. Since Wi > can be arbitrarily large, we have 



lim Pr 

n— >oo 



-logiV«(lf) - nff(Yi) > n(H(Y 2 )-H(Y 1 )) + + 3 7 



Substituting the above equality into ()C.2p . by virtue of the asymptotic normality, it holds that 
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because we can let T3 + 37 < T3 by letting 7 — )• 0. Hence, we have shown the proofs similarly to 
Case I. ■ 
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Appendix D 
Proof of theorem 17.11 

1 ) Direct Part: 

We prove that L\ = Tj+^f is an (H(Yi), e)-achievable, where 7 > is an arbitrary small constant. 
Set M n = e nH(Y 1 )+^fiL 1 _ xhen, obviously we have 

log M n -nH(Y 1 ) 
limsup y= < L\. 



n 



Thus, it is enough to show that there exists an (n, M n ,e n ) code satisfying 



lim sup e n < e. 

From Lemma 17. 11 there exists an (n,M n ,e n ) code that satisfies 
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The last equality is derived from the definition of the mixed source. Then, from Lemma 14. 1\ we 
have 
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because j n is specified in Lemma 14.11 and so 7 > j n holds for sufficiently large n. 
Then, noting that H{Y\) = HQf-z) holds, from the asymptotic normality, we have 
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for i = 1,2. Thus, we have 
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Substituting the above inequality into (ID.lj) . we have 
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where the last equality is derived from the definition of T-j given by ( 17. 2\i . Therefore, Direct Part 
has been proved. 



2) Converse Part: 

Let us assume that L 2 satisfying L 2 < T7 is (H(Yi), e)-achievable. Then we shall show a 
contradiction. Notice that there exists a constant 7 > such that L 2 + 37 < T7 holds. 
Then from the assumption there must exist an (n,M n ,e n ) code such that 

lim sup e n < e, 

v \ogM n -nH{Y l ) 

Inn sup -= < L2, 
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This means that for sufficiently large n, there must exist an (n,M n ,e n ) code such that 

logM n nHiYi] 



< 



+ L 2 + 7. 
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for any 7 > 0. On the other hand, from Lemma 17.21 any (n, M n ,e n ) code satisfies 
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Thus, from (|DT2|) it follows that 
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for sufficiently large n, where the last equality is derived from the definition of the mixed source. 
From Lemma I4JJ we have 



lim inf e n > lim inf > Pr 

n— >oo n— >oo ^- — J 

i=l 

2 

Elim inf Pr 
n— >oo 

i=l 
2 

Elim inf Pr 
r) — ^on 



> 



i=l 



— k>£ 


eJV»(i?) 


-nH{Y 1 ) 








—log 




-nHiYi) 




y/n 




—log 




-nH(Yx) 





> L 2 + 2>yj w(i) 

> L 2 + 2 7 + 7n } 

> L 2 +Sj\w(i) 



(D.3) 



because 7„ is specified in Lemma 14.11 and so 7„ < 7 holds for sufficiently large n. Then, by virtue 
of the asymptotic normality we have 
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because we can let L 2 + 37 < Tj holds by letting 7 — )• and the last equality is from (|7.2p . Thus, 
substituting the above into (|D.3p . it must hold that 

lim inf e n > e. 

n—>oo 

This is a contradiction and we conclude that L 2 satisfying L 2 < Tj is not an (H(Yi), e)-achievable. 



Proofs of Case II and Case III: 
The proofs of Case II and Case III can be shown by the same arguments as the proof of Case I 
of Theorem 17.11 ■ 
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